Interactive user interfaces for minimally invasive telesurgical systems

ABSTRACT

In one embodiment of the invention, a a minimally invasive surgical system is disclosed. The system configured to capture and display camera images of a surgical site on at least one display device at a surgeon console; switch out of a following mode and into a masters-as-mice (MaM) mode; overlay a graphical user interface (GUI) including an interactive graphical object onto the camera images; and render a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

CROSS REFERENCE TO RELATED APPLICATIONS

This non-provisional United States (U.S.) patent application claims thebenefit and is a divisional of U.S. patent application Ser. No.13/775,574, entitled SYSTEMS AND METHODS FOR INTERACTIVE USER INTERFACESFOR ROBOTIC MINIMALLY INVASIVE SURGICAL SYSTEMS, filed on Feb. 25, 2013,now issued as U.S. Pat. No. 9,765,446. U.S. patent application Ser. No.13/775,574 claims the benefit of and is a divisional of U.S. patentapplication Ser. No. 12/189,615 filed on Aug. 11, 2008 by Simon P.Dimaio et al., entitled INTERACTIVE USER INTERFACES FOR ROBOTICMINIMALLY INVASIVE SURGICAL SYSTEMS, now U.S. Pat. No. 8,398,541. U.S.patent application Ser. No. 12/189,615 is a non-provisional applicationand claims the benefit of provisional U.S. Patent Application No.60/954,869 filed on Aug. 9, 2007 by inventors Christopher J. Hasser, etal., entitled ROBOTIC MINIMALLY INVASIVE SURGICAL SYSTEMS, which isincorporated herein by reference. U.S. patent application Ser. No.12/189,615 also claims the benefit of and is a continuation-in-part(CIP) of U.S. patent application Ser. No. 11/447,668 filed on Jun. 6,2006 by Christopher J. Hasser et al entitled ULTRASOUND GUIDANCE FOR ALAPAROSCOPIC SURGICAL ROBOT which is incorporated herein by reference.U.S. patent application Ser. No. 11/447,668 further claims the benefitof U.S. patent application Ser. No. 60/688,013 filed on Jun. 6, 2005.

U.S. patent application Ser. No. 11/447,668 further incorporates byreference U.S. patent application Ser. No. 11/130,471 entitled METHODSAND SYSTEM FOR PERFORMING 3-D TOOL TRACKING BY FUSION OF SENSOR AND/ORCAMERA DERIVED DATA DURING MINIMALLY INVASIVE SURGERY, filed on May 16,2005 by Brian David Hoffman et al.; U.S. Pat. No. 6,659,939 entitledCOOPERATIVE MINIMALLY INVASIVE TELESURGICAL SYSTEM, issued on Dec. 9,2003 to Frederic H. Moll et al.; and U.S. Pat. No. 5,797,900 entitledWRIST MECHANISM FOR SURGICAL INSTRUMENT FOR PERFORMING MINIMALLYINVASIVE SURGERY WITH ENHANCED DEXTERITY AND SENSITIVITY, issued on Aug.25, 1998 to Akhil J. Madhani et al., which are also incorporated hereinby reference.

Furthermore, U.S. Pat. No. 6,522,906 entitled DEVICES AND METHODS FORPRESENTING AND REGULATING AUXILIARY INFORMATION ON AN IMAGE DISPLAY OF ATELESURGICAL SYSTEM TO ASSIST AN OPERATOR IN PERFORMING A SURGICALPROCEDURE, issued on Feb. 18, 2003 to J. Kenneth Salisbury, Jr. et al.;U.S. Pat. No. 6,459,926 entitled REPOSITIONING AND REORIENTATION OFMASTER/SLAVE RELATIONSHIP IN MINIMALLY INVASIVE TELESURGERY, issued onOct. 1, 2006 to William C. Nowlin et al.; U.S. Pat. No. 6,493,608entitled ASPECTS OF A CONTROL SYSTEM OF A MINIMALLY INVASIVE SURGICALAPPARATUS, issued on Dec. 10, 2002 to Gunter D. Niemeyer; U.S. Pat. No.6,799,065 entitled IMAGE SHIFTING APPARATUS AND METHOD FOR A TELEROBOTICSYSTEM, issued on Sep. 28, 2004 to Gunter D. Niemeyer; and U.S. Pat. No.6,714,939 entitled MASTER HAVING REDUNDANT DEGREES OF FREEDOM, issued onMar. 30, 2004 to Salisbury et al. are all incorporated herein byreference.

FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

The assignees of this United States (U.S.) patent application elect toretain the rights in this invention. This invention was made withgovernment support under EEC9731748 and EEC0646678 awarded by theNational Science Foundation and under RR019159 awarded by the NationalInstitutes of Health. The government has certain rights in theinvention.

FIELD OF INVENTION

Aspects of the invention are related to user interfaces for a surgeon'sworkstation in robotic surgical systems.

BACKGROUND OF INVENTION

Minimally invasive robotic surgical systems, such as the da Vinci®Surgical System, are manufactured by Intuitive Surgical, Inc., ofSunnyvale, Calif. The Johns Hopkins University Engineering ResearchCenter for Computer-Integrated Surgical Systems and Technology(ERC-CISST) conducts research in aspects of minimally invasive surgicalsystems.

The number of robotic arms available in minimally invasive roboticsurgical systems has been slowly increasing to support additionalrobotic surgical tools over a patient. Additionally, some more recentrobotic surgical tools have a greater number of controllable features.Unfortunately, a surgeon has only a pair of eyes, hands and feet toselect and control the greater number of tools and controllable featuresof the robotic surgical tools.

SUMMARY OF INVENTION

The embodiments of the invention are summarized by the claims thatfollow below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a top view of an operating room employing alaparoscopic ultrasound robotic surgical system utilizing aspects of theembodiments of the invention.

FIG. 2 illustrates a block diagram of a laparoscopic ultrasound roboticsurgical system utilizing aspects of the embodiments of the invention.

FIG. 3 illustrates a laparoscopic ultrasound probe utilizing aspects ofthe embodiments of the invention.

FIG. 4 illustrates a flow diagram of a method for training a LUS roboticsurgical system to robotically move a LUS probe in a trained manner uponcommand, utilizing aspects of the embodiments of the invention.

FIG. 5 illustrates a flow diagram of a method for generating a clickablethumbnail image that allows a user to command that a LUS probe beautomatically moved to a position and orientation from which the imagewas captured, utilizing aspects of the embodiments of the invention.

FIG. 6 illustrates a flow diagram of a method for automatically moving sLUS probe to a position and orientation associated with a clickablethumbnail image, utilizing aspects of the embodiments of the invention.

FIG. 7 illustrates a flow diagram of a method for robotically assistedneedle guidance to a marked lesion of a cancerous structure, utilizingaspects of the embodiments of the invention.

FIG. 8 illustrates a perspective view of a 3D ultrasound image of ananatomic structure in a camera reference frame with selectable 2D imageslices as used in a medical robotic system utilizing aspects of theembodiments of the invention.

FIG. 9 illustrates a perspective view of a 3D camera view of an anatomicstructure in a camera reference as used in a medical robotic systemutilizing aspects of the embodiments of the invention.

FIG. 10 illustrates a perspective view of a frontal 2D slice of a 3Dultrasound view of an anatomic structure that overlays a 3D camera viewof the anatomic structure, as displayable in a medical robotic systemutilizing aspects of the embodiments of the invention.

FIG. 11 illustrates a perspective view of an inner 2D slice of a 3Dultrasound view of an anatomic structure that overlays a 3D camera viewof the anatomic structure, as displayable in a medical robotic systemutilizing aspects of the embodiments of the invention.

FIG. 12 is a diagrammatic physical view of a subsystem architecture fora surgical assistant workstation.

FIG. 13 is a diagrammatic view of a surgical assistant workstation forteleoperated surgical robots architecture.

FIG. 14 is a diagrammatic view of an illustrative data flow.

FIG. 15 is a diagrammatic view of a subsystem architecture (logicalview).

FIG. 16 is a diagrammatic view of basic three dimensional (3D) pointerinteraction logic—event handling.

FIG. 17 is a diagrammatic view of basic 3D pointer interactionlogic—move event.

FIG. 18 is a diagrammatic view of basic 3D pointer interactionlogic—grab event.

FIG. 19 is a diagrammatic view of a subsystem architecture (processview).

FIG. 20 is a diagrammatic view of a subsystem architecture (developmentview).

FIG. 21 is a diagrammatic view of a 3D interface display for a surgeonconsole in a minimally invasive surgical system.

FIGS. 22A-22C are exemplary diagrammatic views of invoking the graphicaluser interface to overlay menu systems, icons, a pointer, and aflashlight view of images onto the captured camera images displayed inthe 3D interface display of the surgeon console.

FIG. 23 is an exemplary diagrammatic view of a pointer being used toselect menu items to select context sensitive menu items from a menuoverlaid onto the captured camera images adjacent a surgical instrument.

FIGS. 24A-24D are exemplary diagrammatic views of invoking the graphicaluser interface to overlay menu systems, icons, a pointer, and a medicalimage volume onto the captured camera images displayed in the 3Dinterface display of the surgeon console.

FIGS. 25A-25E are exemplary diagrammatic views of manipulating a medicalimage volume and selecting menu items overlaid on the captured cameraimages displayed in the 3D interface display of the surgeon console.

FIGS. 26A-26B are exemplary diagrammatic views of selecting menu itemsto display sagittal image slices of the medical image volume overlaid onthe captured camera images displayed in the 3D interface display of thesurgeon console.

FIGS. 27A-27B are exemplary diagrammatic views of selecting a axialslice plane and manipulating the axial slice plane to display differentimage slices of the medical image volume overlaid on the captured cameraimages displayed in the 3D interface display of the surgeon console.

FIG. 28 is an exemplary diagrammatic view of menu selection to close amedical image volume.

FIG. 29 is an exemplary diagrammatic view of boundaries of a virtualfixture overlaid onto the camera images of the surgical site in the 3Dinterface display of the surgeon console.

DETAILED DESCRIPTION

In the following detailed description of the embodiments of theinvention, numerous specific details are set forth in order to provide athorough understanding of the invention. However, the embodiments of theinvention may be practiced without these specific details. In otherinstances well known methods, procedures, components, and circuits havenot been described in detail so as not to unnecessarily obscure aspectsof the embodiments of the invention.

Introduction

Various embodiments of a minimally invasive surgical master/slaverobotic system allow ultrasonic image display, image manipulation,supervisor/trainee master consoles, automatic movement limitation, andinterchangeable slave consoles. For example, U.S. patent applicationSer. No. 11/447,668 entitled ULTRASOUND GUIDANCE FOR A LAPAROSCOPICSURGICAL ROBOT to which priority is claimed, describes a minimallyinvasive surgical robotic system with a laparoscopic ultrasonic robotictool.

FIG. 1 illustrates, as an example, a top view of an operating roomemploying a robotic surgical system. The robotic surgical system in thiscase is a laparoscopic ultrasound robotic surgical system 100 includinga console (“C”) (also may be referred to herein as a surgeon console,master console, master surgeon console, or surgical console) utilized bya surgeon (“S”) while performing a minimally invasive diagnostic orsurgical procedure with assistance from one or more assistants (“A”) ona patient (“P”) who is reclining on an operating table (“O”).

The console C includes a master display 104 (also referred to herein asa display screen or display device) for displaying one or more images ofa surgical site within the patient as well as perhaps other informationto the surgeon. Also included are master input devices 107 and 108 (alsoreferred to herein as master manipulators or master tool manipulators(MTM), master grips, hand control devices), one or more foot pedals 105and 106, a microphone 103 for receiving voice commands from the surgeon,and a processor 102. The master input devices 107 and 108 may includeany one or more of a variety of input devices such as joysticks, gloves,trigger-guns, hand-operated controllers, or the like. The processor 102may be a computer or a part of a computer that may be integrated intothe surgeon console or otherwise connected to the surgeon console in aconventional manner.

The surgeon performs a minimally invasive surgical procedure bymanipulating the master input devices 107 and 108 so that the processor102 causes their respectively associated slave arms 121 and 122 (alsoreferred to herein as slave manipulators or slave robots) of the patientside cart (PSC) 120 to manipulate their respective removeably coupledand held surgical instruments 138 and 139 (also referred to herein astools or minimally invasive surgical instruments) accordingly, while thesurgeon views three-dimensional (“3D”) images of the surgical site onthe master display 104.

The tools 138 and 139, in one embodiment of the invention, are IntuitiveSurgical Inc.'s proprietary ENDOWRIST™ articulating instruments, whichare modeled after the human wrist so that when added to the motions ofthe robot arm holding the tool, they allow a full six degrees of freedomof motion, which is comparable to the natural motions of open surgery.Additional details on such tools may be found in U.S. Pat. No. 5,797,900entitled WRIST MECHANISM FOR SURGICAL INSTRUMENT FOR PERFORMINGMINIMALLY INVASIVE SURGERY WITH ENHANCED DEXTERITY AND SENSITIVITY,issued on Aug. 25, 1998 to Akhil J. Madhani et al. which is incorporatedherein by this reference. At the operating end of each of the tools 138and 139 is a manipulatable end effector such as a clamp, grasper,scissor, stapler, blade, needle, or needle holder.

The master display 104 is a high-resolution stereoscopic video displaydevice. In one embodiment of the invention, the high-resolutionstereoscopic video display device is formed of two progressive scancathode ray tubes (“CRTs”). In another embodiment of the invention, thehigh-resolution stereoscopic video display device is formed of twoliquid crystal display (“LCDs”) devices. The system offers higherfidelity than polarization, shutter eyeglass, or other techniques. Eacheye views a separate display presenting the left or right eyeperspective, through an objective lens and a series of mirrors. Thesurgeon sits comfortably and looks into this display throughout surgery,making it an ideal place for the surgeon to display and manipulate 3-Dintraoperative imagery.

A stereoscopic endoscope 140 (also referred to herein as an endoscopiccamera) provides right and left camera views to the processor 102 sothat it may process the information according to programmed instructionsand cause it to be displayed on the master display 104. A laparoscopicultrasound (“LUS”) probe 150 provides two-dimensional (“2D”) ultrasoundimage slices of an anatomic structure to the processor 102 so that theprocessor 102 may generate a 3D ultrasound computer model of theanatomic structure and cause the 3D computer model (or alternatively, 2D“cuts” of it) to be displayed on the master display 104 as an overlay tothe endoscope derived 3D images or within a picture-in-picture (“PIP”)in either 2D or 3D and from various angles and/or perspectives accordingto surgeon or stored program instructions.

Each of the tools 138 and 139, as well as the endoscope 140 and LUSprobe 150, is preferably inserted through a cannula or trocar (notshown) or other tool guide into the patient so as to extend down to thesurgical site through a corresponding minimally invasive incision suchas Incision 166. Each of the slave arms 121-124 is conventionally formedof linkages which are coupled together and manipulated through motorcontrolled joints (also referred to as “active joints”). Setup arms (notshown) comprising linkages and setup joints are used to position theslave arms 121-124 vertically and horizontally so that their respectivesurgical related instruments may be coupled for insertion into thecannulae.

The number of surgical tools used at one time and consequently, thenumber of slave arms being used in the system 100 will generally dependon the diagnostic or surgical procedure and the space constraints withinthe operating room, among other factors. If it is necessary to changeone or more of the tools being used during a procedure, the Assistantmay remove the tool no longer being used from its slave arm, and replaceit with another tool, such as a minimally invasive surgical tool 131,from a tray (“T”) in the operating room.

Preferably, the master display 104 is positioned near the surgeon'shands so that it will display a projected image that is oriented so thatthe surgeon feels that he or she is actually looking directly down ontothe surgical site. To that end, an image of the tools 138 and 139preferably appear to be located substantially where the surgeon's handsare located even though the observation points (i.e., that of theendoscope 140 and LUS probe 150) may not be from the point of view ofthe image.

In addition, the real-time image is preferably projected into aperspective image such that the surgeon can manipulate the end effectorof a tool, 138 or 139, through its associated master input device, 107or 108, as if viewing the workspace in substantially true presence. Bytrue presence, it is meant that the presentation of an image is a trueperspective image simulating the viewpoint of an operator that isphysically manipulating the tools. Thus, the processor 102 transformsthe coordinates of the tools to a perceived position so that theperspective image is the image that one would see if the endoscope 140was looking directly at the tools from a surgeon's eye-level during anopen cavity procedure.

The processor 102 performs various functions in the system 100. Onefunction that it performs is to translate and transfer the mechanicalmotion of master input devices 107 and 108 to their associated slavearms 121 and 122 through control signals over bus 110 so that thesurgeon can effectively manipulate their respective tools 138 and 139.Another function of the processor 102 is to implement the variousmethods and functions described herein, including providing a roboticassisted LUS capability.

Although described as a processor, it is to be appreciated that theprocessor 102 may be implemented in practice by any combination ofhardware, software and firmware. Also, its functions as described hereinmay be performed by one unit, or divided up among different components,each of which may be implemented in turn by any combination of hardware,software and firmware. Program code or instructions for the processor102 to implement the various methods and functions described herein maybe stored in processor readable storage media, such as memory (e.g.,memory 240 illustrated in FIG. 2).

Prior to performing a minimally invasive surgical procedure, ultrasoundimages captured by the LUS probe 150, right and left 2D camera imagescaptured by the stereoscopic endoscope 140, and end effector positionsand orientations as determined using kinematics of the slave arms121-124 and their sensed joint positions, are calibrated and registeredwith each other.

In order to associate the ultrasound image with the rest of the surgicalenvironment, both need to be expressed in the same coordinate frame.Typically, the LUS probe 150 is either labeled with markers and trackedby a tracking device such as the OPTORAK® position sensing systemmanufactured by Northern Digital Inc. of Ontario, Canada, or held by arobot with precise joint encoders. Then the rigid transformation betweenthe ultrasound image and the frame being tracked is determined (which istypically referred to as the ultrasound calibration).

For example, using the OPTOTRAK® frame for the ultrasound calibration,the ultrasound image generated by the LUS probe 150 is calibrated to anOPTOTRAK® rigid body using an AX=XB formulation. “AX=XB” is a rubric fora class of calibration/registration problem commonly encountered incomputer vision, surgical navigation, medical imaging, and robotics. Themathematical techniques are well known. See, e.g., E. Boctor, A.Viswanathan, M. Chioti, R. Taylor, G. Fichtinger, and G. Hager, “A NovelClosed Form Solution for Ultrasound Calibration,” InternationalSymposium on Biomedical Imaging, Arlington, Va., 2004, pp. 527-530.

“A” and “B” in this case, are transformations between poses of theOPTOTRAK® rigid body (A) and the ultrasound image (B). Thus, “X” is thetransformation from the ultrasound image to the rigid body.

To perform the ultrasound calibration, the LUS probe 150 may be placedin three known orientations defined by the AX=XB calibration phantom.The ultrasound image frame may then be defined by three fiducials whichappear in each of the three poses. The three poses allow three relativetransformations based on OPTOTRAK® readings (A) and three relativetransformations based on the ultrasound images (B) for the AX=XBregistration.

Camera calibration is a common procedure in computer visionapplications. As an example, in order to determine the intrinsic andextrinsic parameters of the endoscope 140, a checkerboard phantom with amulti-plane formulation may be provided by the Caltech's cameracalibration toolbox. To construct the phantom, OPTOTRAK® markers areadded to a typical checkerboard video calibration phantom, and eachcorner of the checkerboard is digitized using a calibrated OPTOTRAK®pointer. Thus, the corner positions may be reported with respect to theOPTOTRAK®.

The calibration may then be performed by placing the phantom in view ofthe endoscope 140 in several dozen orientations, and recording bothstereo image data and OPTOTRAK® readings of the four checkerboardcorners. The images may then be fed into the calibration toolbox, whichdetermines the intrinsic and extrinsic camera parameters, as well as the3D coordinates of the grid corners in the camera frame. Thesecoordinates may then be used with the OPTOTRAK® readings to perform apoint-cloud to point-cloud registration between the endoscope 140 rigidbody and camera frame.

The processor/controller 102 is configured to use the robot kinematicsto report a coordinate frame for the LUS probe 150 tip relative to theendoscope 140. However, due to inaccuracies in the setup joint encoders,both of these coordinate frames may be offset from their correct values.Thus, it may be necessary to register the offsets between the realcamera frame of the endoscope 140 and the camera frame calculated fromthe kinematics as well as between the real and kinematic LUS probe 150frames. With this complete, the kinematics may be used in place of theOPTOTRAK® readings to determine ultrasound image overlay placement.

If the position of the endoscope 140 doesn't overly change, a constanttransformation may be assumed between the kinematic tool tip and thelaparoscopic OPTOTRAK® rigid body. Using an AX=XB formulation, the LUSprobe 150 may be moved, for example, to several positions, and thestatic offset between the tool tip and OPTOTRAK® rigid body registered.Knowing this offset, the endoscope 140 offset may be calculateddirectly:C _(CD) =D _(LusD)(C _(LusUrb))⁻¹ T _(OUrb)(T _(OErb))⁻¹ F _(CErb)  (1)where C_(CD) is the camera offset from the real endoscope 140 (alsoreferred to herein simply as the “camera”) frame to the camera framecalculated from the kinematics, F_(CErb) is the transformation from thecamera to the endoscope rigid body, T_(OUrb)*(T_(OErb))⁻¹ is thetransformation from the camera rigid body to the LUS rigid body,C_(LusUrb) is the transformation from the LUS rigid body to thekinematic ultrasound tool tip, and D_(LusD) is the reading from theprocessor/controller 102 giving the transformation from the kinematicultrasound tool tip to a fixed reference point associated with the slavearms 121-124.

However, registration may be redone each time the camera is moved. Forintra-operative, the registration may be better performed using videotracking of a visual marker on the LUS probe 150 instead of theOPTOTRAK® readings. Thus, if the camera were moved while using tooltracking, the registration can be corrected on the fly as the tool istracked. For additional details on tool tracking, see, e.g., U.S. patentapplication Ser. No. 11/130,471 entitled METHODS AND SYSTEM FORPERFORMING 3-D TOOL TRACKING BY FUSION OF SENSOR AND/OR CAMERA DERIVEDDATA DURING MINIMALLY INVASIVE SURGERY, filed on May 16, 2005 by BrianDavid Hoffman et al., which is incorporated herein by reference. Inaddition to, or alternatively, manual registration of ultrasound andcamera images may be performed using conventional grab, move and rotateactions on a 3D ultrasound computer model of an anatomic structure, sothat the computer model is properly registered over a camera model ofthe anatomic structure in the master display 104.

Slave arms 123 and 124 may manipulate the endoscope 140 and LUS probe150 in similar manners as slave arms 121 and 122 manipulate tools 138and 139. When there are only two master input devices in the system,however, such as master input devices 107 and 108 in the system 100, inorder for the surgeon to manually control movement of either theendoscope 140 or LUS probe 150, it may be required to temporarilyassociate one of the master input devices 107 and 108 with the endoscope140 or the LUS probe 150 that the surgeon desires manual control over,while its previously associated tool and slave manipulator are locked inposition.

FIG. 2 illustrates, as an example, a block diagram of the LUS roboticsurgical system 100. In this system, there are two master input devices107 and 108. The master input device 107 controls movement of either atool 138 or a stereoscopic endoscope 140, depending upon which mode itscontrol switch mechanism 211 is in, and master input device 108 controlsmovement of either a tool 139 or a LUS probe 150, depending upon whichmode its control switch mechanism 231 is in.

The control switch mechanisms 211 and 231 may be placed in either afirst or second mode by a surgeon using voice commands, switchesphysically placed on or near the master input devices 107 and 108, footpedals 105 and 106 on the console, or surgeon selection of appropriateicons or other graphical user interface selection means displayed on themaster display 104 or an auxiliary display (not shown).

When control switch mechanism 211 is placed in the first mode, it causesmaster controller 202 to communicate with slave controller 203 so thatmanipulation of the master input 107 by the surgeon results incorresponding movement of tool 138 by slave arm 121, while the endoscope140 is locked in position. On the other hand, when control switchmechanism 211 is placed in the second mode, it causes master controller202 to communicate with slave controller 233 so that manipulation of themaster input 107 by the surgeon results in corresponding movement ofendoscope 140 by slave arm 123, while the tool 138 is locked inposition.

Similarly, when control switch mechanism 231 is placed in the firstmode, it causes master controller 222 to communicate with slavecontroller 223 so that manipulation of the master input 108 by thesurgeon results in corresponding movement of tool 139 by slave arm 122.In this case, however, the LUS probe 150 is not necessarily locked inposition. Its movement may be guided by an auxiliary controller 242according to stored instructions in memory 240. The auxiliary controller242 also provides haptic feedback to the surgeon through master input108 that reflects readings of a LUS probe force sensor 247. On the otherhand, when control switch mechanism 231 is placed in the second mode, itcauses master controller 222 to communicate with slave controller 243 sothat manipulation of the master input 222 by the surgeon results incorresponding movement of LUS probe 150 by slave arm 124, while the tool139 is locked in position.

Before switching back to the first or normal mode, the master inputdevice 107 or 108 is preferably repositioned to where it was before theswitch to the second mode of Control Switch 211 or 231, as the case maybe, or kinematic relationships between the master input device 107 or108 and its respective tool slave arm 121 or 122 is readjusted so thatupon switching back to the first or normal mode, abrupt movement of thetool 138 or 139 does not occur. For additional details on controlswitching, see, e.g., U.S. Pat. No. 6,659,939 entitled COOPERATIVEMINIMALLY INVASIVE TELESURGICAL SYSTEM, issued on Dec. 9, 2003 toFrederic H. Moll et al., which is incorporated herein by this reference.

The auxiliary controller 242 also performs other functions related tothe LUS probe 150 and the endoscope 140. It receives output from a LUSprobe force sensor 247, which senses forces being exerted against theLUS probe 150, and feeds the force information back to the master inputdevice 108 through the master controller 222 so that the surgeon mayfeel those forces even if he or she is not directly controlling movementof the LUS probe 150 at the time. Thus, potential injury to the patientis minimized since the surgeon has the capability to immediately stopany movement of the LUS probe 150 as well as the capability to take overmanual control of its movement.

Another key function of the auxiliary control 242 is to cause processedinformation from the endoscope 140 and the LUS probe 150 to be displayedon the master display 104 according to user selected display options. Aswill be described in more detail below, such processing includesgenerating a 3D ultrasound image from 2D ultrasound image slicesreceived from the LUS probe 150 through an Ultrasound processor 246,causing either 3D or 2D ultrasound images corresponding to a selectedposition and orientation to be displayed in a picture-in-picture windowof the master display 104, and causing either 3D or 2D ultrasound imagesof an anatomic structure to overlay a camera captured image of theanatomic structure being displayed on the master display 104.

Although shown as separate entities, the master controllers 202 and 222,slave controllers 203, 233, 223, and 243, and auxiliary controller 242are preferably implemented as software modules executed by the processor102, as well as certain mode switching aspects of the control switchmechanisms 211 and 231. The Ultrasound processor 246 and Video processor236, on the other hand, are separate boards or cards typically providedby the manufacturers of the LUS probe 150 and endoscope 140 that areinserted into appropriate slots coupled to or otherwise integrated withthe processor 102 to convert signals received from these image capturingdevices into signals suitable for display on the master display 104and/or for additional processing by the auxiliary controller 242 beforebeing displayed on the master display 104.

FIG. 3 illustrates a side view of one embodiment of the laparoscopicultrasound (LUS) probe 150. The LUS probe 150 is a dexterous tool withpreferably two distal degrees of freedom, permitting reorientation oflaparoscopic ultrasound (LUS) sensor 301 through, for example,approximately ±80° in distal “pitch” and “yaw”, and ±240° in “roll”about a ball joint type, pitch-yaw mechanism 311 (functioning as andalso referred to herein as a “Wrist” mechanism). Opposing pairs of DriveRods or cables (not shown) physically connected to a proximal end of theLUS sensor 301 and extending through an internal passage of elongatedshaft 312 mechanically control pitch and yaw movement of the LUS sensor301 using conventional push-pull type action. This flexibility of theLUS probe 150 (provided by the pitch/yaw wrist mechanism) is especiallyuseful in optimally orienting the LUS probe 150 for performingultrasonography on an anatomic structure during a minimally invasivesurgical procedure.

The LUS sensor 301 captures 2D ultrasound slices of a proximate anatomicstructure, and transmits the information back to the processor 102through LUS cable 304. Although shown as running outside of theelongated shaft 312, the LUS cable 304 may also extend within it. Aclamshell sheath 321 encloses the elongated shaft 312 and LUS cable 304to provide a good seal passing through a cannula 331 (or trocar).Fiducial marks 302 and 322 are placed on the LUS sensor 301 and thesheath 321 for video tracking purposes.

A force sensing capability is provided by strain gauges 303 whichprovide direct feedback of how hard the LUS probe 150 is pushing on astructure being sonographed, supplementing whatever limited feedback isavailable from joint motor torques. Potential uses of this informationinclude: providing a redundant safety threshold check warning thesurgeon or preventing motion into the structure if forces get too great;providing the surgeon with an approved haptic appreciation of how hardhe or she is pushing on a structure; and possibly permitting somemeasure of compensation for unmodeled deflections of the pitch-yaw orwrist mechanism 311 which are not detected for some reason by jointposition sensors or encoders. The strain gauges 303 in this case servethe function of the LUS probe force sensor 247 as previously describedin reference to FIG. 2.

Robotic assisted LUS has the potential to reduce variability in theultrasound images produced, compared to freehand scanning, and canreduce operator workload and difficulty. Behaviors as simple as rockingthe LUS probe 150 back and forth can maintain an updated 3D ultrasoundimage without operator intervention. More complicated behaviors caninclude movement of the LUS probe 150 along the surface of a targetanatomical structure in a methodical pattern to generate a full image ofthe target, or reliably returning to a previously scanned probe locationand orientation.

FIG. 4 illustrates, as an example, a flow diagram of a method fortraining the auxiliary controller 242 (i.e., providing it with storedinstructions) to cause the LUS probe 150 to be robotically moved in thetrained manner upon command, in order to capture a sequence of 2Dultrasound image slices of an anatomic structure, which are used by theauxiliary controller 242 to generate a 3D computer model of thestructure. Prior to performing the training, the control switchmechanism 231 is placed in its second mode so that the surgeon may movethe LUS probe 150 for training purposes by manipulating the master inputdevice 108. After performing training, the control switch mechanism 231is then placed back into its first or normal mode so that the surgeonmay manipulate the tool 139 to perform a minimally invasive surgicalprocedure using the master input device 108.

In process 401, the training module is initially idle (i.e., it is notbeing executed by the processor 102). In process 402, the processor 102(or a training module agent running in the background) may periodicallycheck whether a start of training indication is received. Alternatively,the start of training indication may act as an interrupt which initiatesrunning of the training module. The start of training indication may beinitiated by a surgeon through a recognized voice command, selection ofa training option on a graphical user interface displayed on the masterdisplay 104, a switch mechanism that may physically be located on thecorresponding master Control Input 108 or other convenient locationaccessible to the surgeon, or any other conventional means.

After the start of training indication is detected, in process 403, thetraining module records or stores the current LUS probe 150 position andorientation, and periodically (or upon surgeon command) continues to doso by looping around processes 403 and 404 until a stop trainingindication is detected or received. The stop training indication in thiscase may also be initiated by the surgeon in the same manner as thestart of training indication, or it may be initiated in a different, butother conventional manner. After the stop training indication isdetected or received, a last position and orientation of the LUS probe150 is recorded or stored.

Between the start and stop of training, the surgeon moves the LUS probe150 and the processor 102 stores its trajectory of points andorientations so that they may be retraced later upon command. In onetype of training, the surgeon moves the LUS probe 150 back and forthnear an anatomic structure in order to capture a sequence of 2Dultrasound image slices from which a 3D version (or computer model) ofthe anatomic structure may be rendered by the processor 102. In anothertype of training, the surgeon move the LUS probe 150 once or more timesalong the surface of the anatomic structure in order to capture adifferent sequence of 2D ultrasound image slices from which a 3D version(or computer model) of the anatomic structure may be rendered by theprocessor 102.

Although described as recording the positions and orientations of theLUS probe 150, in practice, the active joint positions of its slave arm124 are stored instead since their measurements are directly obtainablethrough encoders attached to each of the joints and their positionscorrespond to the LUS probe 150 positions and orientations.

After storing the trajectory of positions and orientations of the LUSprobe 150 in the memory 240, the trajectory is then associated with ameans for the surgeon to command the auxiliary controller 242 to movethe LUS probe 150 in the desired fashion. For example, the trajectorymay be associated with a voice command which upon its detection, theauxiliary controller 242 causes the slave arm 124 to move the LUS probe150 back and forth along the stored trajectory of positions andorientations. Likewise, the trajectory may also be associated with auser selectable option on a graphical user interface displayed on themaster display 104, or it may be associated with a switch mechanism suchas a button or unused control element on the master input device 108. Itmay also be associated with the depression of the foot pedal 106, sothat the auxiliary controller 242 causes the slave arm 124 to move theLUS probe 150 back and forth along the stored trajectory of positionsand orientations as long as the foot pedal 106 is being depressed, andstops such motion once the surgeon takes his or her foot off the footpedal 106.

FIG. 5 illustrates, as an example, a flow diagram of a method forgenerating clickable thumbnail images corresponding to LUS probe 150positions and orientations that are stored in memory 240, so that whenthe surgeon clicks on one of the thumbnail images, the auxiliarycontroller 242 causes the slave arm 124 to move the LUS probe 150 to itsstored position and orientation. This allows the surgeon to move the LUSprobe 150 to see different views of an anatomic structure while thecontrol switch mechanism 231 is in its first or normal mode. Thus, thesurgeon can continue to perform a minimally invasive surgical procedureby manipulating tool 139 using the master input device 108. The methodmay then be combined with that described in reference to FIG. 4 in orderto generate a sequence of 2D ultrasound image slices starting from thatposition and orientation, from which the auxiliary controller 242 maygenerate a 3D computer model rendition of the anatomic structure.

Prior to performing the method, however, the control switch mechanism231 is placed in its second mode so that the surgeon may move the LUSprobe 150 into the desired positions and orientations by manipulatingthe master input device 108. After generating the clickable thumbnailimages, the control switch mechanism 231 is then placed back into itsfirst or normal mode so that the surgeon may manipulate the tool 139 toperform the minimally invasive surgical procedure using the master inputdevice 108.

In process 501, the auxiliary controller 242 receives a snapshot commandfrom the surgeon. The snapshot command may be, for example, a voicecommand, graphical user interface selection, or switch position. Inprocess 502, the auxiliary controller 242 causes the LUS probe 150 tocapture a 2D ultrasound image slice, and in process 503, a thumbnail ofthe image is generated. The thumbnail in this case may include a simpleJPEG or GIF file of the captured image. In process 504, the currentposition and orientation of the LUS probe 150 is stored in memory 240along with information of its association with the thumbnail. In process505, a clickable version of the thumbnail is displayed on the masterdisplay 104, so that the surgeon may command the auxiliary controller242 to cause the LUS probe to be positioned and oriented at the storedposition and orientation at any time upon clicking with his or her mouseor other pointing device on the clickable thumbnail. The surgeon maythen move the LUS probe 150 to other positions and/or orientations, andrepeat processes 501-505 to generate additional thumbnail images.

FIG. 6 illustrates, as an example, a flow diagram of a method forautomatically moving the LUS probe 150 to a position and orientationassociated with a clickable thumbnail upon command to do so by a surgeonwhile performing a minimally invasive surgical procedure using tool 139.In process 601, the clicking of a thumbnail generated by the methoddescribed in reference to FIG. 5 is detected by, for example, aconventional interrupt handling process. Upon such detection, in process602, the auxiliary controller 242 is instructed by, for example, storedinstructions corresponding to the interrupt handling process, toretrieve the position and orientation stored in memory 240 which isassociated with the thumbnail. The auxiliary controller 242 then causesthe LUS probe 150 to move to that position and orientation byappropriately controlling slave arm 124 in process 603. Thus, thesurgeon is able to move the LUS probe 150 to a desired position withouthaving to change modes of the control switch mechanism 231 and haltoperation of the tool 139 until the LUS probe 150 is moved.

Virtual Fixtures

The processor 102 may generate a virtual fixture, such as a guidancevirtual fixture or a forbidden region virtual fixture. To generate thevirtual fixture, local kinematic constraints on the slave armmanipulating the tool may be specified by providing a table ofconstraints. Generally, a virtual fixture can limit movement of asurgical instrument or tool. For example, a guidance virtual fixture maybe generated to assist in electronically constraining a tool to travelover a predetermined path. A forbidden region virtual fixture may begenerated to

A variety of types and shapes of virtual fixtures may be generated tolimit movement of a minimally invasive surgical tool such as virtualplanes, virtual chamfers, virtual springs, detents, etc. With thesevirtual fixtures based on position in mind, virtual dampers may begenerated by adding velocity terms.

FIG. 7 illustrates, as an example, a flow diagram of a method forrobotically assisted needle guidance and penetration into a markedlesion of a cancerous structure, which allows appreciation for theaspects of robotic assisted LUS described herein. In process 701, aselected 2D ultrasound image slice view of a cancerous structure such asa liver is displayed at the proper depth on the master display 104 as anoverlay to a 3D camera view of the cancerous structure. The selected 2Dultrasound image slice view may be a frontal view or an inner slice viewas taken from a previously generated 3D ultrasound computer model of thecancerous structure.

As an example clarifying the process 701, FIG. 8 illustrates asimplified perspective view of a 3D ultrasound computer model 800 of thecancerous structure, which has been generated, for example, using themethod described in reference to FIG. 4, and has been translated intothe camera reference frame (EX, EY, EZ). FIG. 9, on the other hand,illustrates a simplified perspective view of a 3D camera view 900 of thecancerous structure as taken by the stereoscopic endoscope 140. If thesurgeon selects a frontal slice 801 of the 3D ultrasound computer model800 to be viewed as an overlay to the 3D camera view 900, then theoverlay will appear as shown in FIG. 10. On the other hand, if thesurgeon selects one of the inner slices 802-804 of the 3D ultrasoundcomputer model 800, such as inner slice 803, to be viewed as an overlayto the 3D camera view 900, then the overlay will appear as shown in FIG.11 with the 2D ultrasound image slice 803 displayed at the proper depth.To avoid confusion, the portion of the 3D camera view above that depthis made transparent.

Alternatively, the surgeon may manually control movement of the LUSprobe 150 so that 2D ultrasound image slices captured by it appear asemanating in proper perspective and direction from the 3D camera imageof the LUS probe 150 in the master display 104. Preferably, the emanated2D image slices being displayed in the master display 104 do not occludethe anatomic structure being probed. This manual approach may beparticularly useful to the surgeon for quickly spotting lesions in theanatomic structure.

In process 702, the surgeon marks lesions on the cancerous structuredisplayed as a result of process 701. Each marked lesion is preferablymarked using a designated color in order to clearly show that thesurgeon has already identified it, thereby avoiding double counting. Thelocation in the camera reference frame (EX, EY, EZ) of each markedlesion is stored in memory 240, and in process 703, the processor 102determines an optimal needle tip path to that location.

In process 704, the processor 102 generates a virtual fixture to helpguide the needle to the marked lesion. To generate the virtual fixture,local kinematic constraints on the slave arm manipulating the needletool may be specified by providing a table of constraints of the form:({right arrow over (x)}−{right arrow over (x)}₀)^(T) A _(K)({right arrowover (x)}−{right arrow over (x)}₀)+{right arrow over (b)}_(K)({rightarrow over (x)}−{right arrow over (x)}₀)≤c  (2)where {right arrow over (x)} represents, in simplified terms, thecurrent 6 DOF kinematic pose of a master arm, or, in more general terms,a parameterization of a Cartesian pose F linearized about some nominalpose F₀ so that ({right arrow over (x)}−{right arrow over (x)}₀)˜F₀ ⁻¹F. The tables are to be updated periodically based on visual feedback,user interaction, etc.

As can be appreciated, equation (2) can be easily checked and enforced.

Similarly, a simple table-driven interface for surgeon interactionforces can be implemented approximately as follows:

$\quad\begin{matrix}{\left. {\left. \overset{\rightarrow}{f}\leftarrow 0 \right.;\left. y\leftarrow{\overset{\rightarrow}{x} - {\overset{\rightarrow}{x}}_{0}} \right.;}{{for}\mspace{14mu} k}\leftarrow{1\mspace{14mu}{to}\mspace{14mu} N\mspace{14mu}{do}} \right.\mspace{20mu}{\left\{ {\left. ɛ\leftarrow{{{\overset{\rightarrow}{y}}^{T}C_{K}\overset{\rightarrow}{y}} + {{\overset{\rightarrow}{d}}_{K}\overset{\rightarrow}{y}} - e_{K}} \right.;\mspace{40mu}{{{if}\mspace{14mu} ɛ} > {0\mspace{14mu}{then}\mspace{14mu}\left\{ {\left. \overset{\rightarrow}{g}\leftarrow{2C_{K}\overset{\rightarrow}{y}{\overset{\rightarrow}{d}}_{k}} \right.;\left. \overset{\rightarrow}{f}\leftarrow{\overset{\rightarrow}{f} + {{f(ɛ)}{\overset{\rightarrow}{g}/{\overset{\rightarrow}{g}}}}} \right.;} \right\}}};}\mspace{20mu} \right\};}} & (3)\end{matrix}$

output {right arrow over (f)} (after limiting & spacing)

where ε corresponds, roughly, to a distance from a surface in statespace and the function f (ε) corresponds to a (non-linear) stiffness.

The above formulation suffices to support a variety of virtual chamfers,virtual springs, detents, etc. The formulation can be easily extended tovirtual dampers by adding velocity terms.

Now, more particularly, in the present case where it is desired to helpaim an injection needle at a target in a live ultrasound image, let:

$\begin{matrix}\begin{matrix}{{\overset{\rightarrow}{P}}_{TROCAR} = {{position}\mspace{14mu}{where}\mspace{14mu}{needle}\mspace{14mu}{enters}\mspace{14mu}{patient}}} \\{= {{{}_{}^{}{}_{}^{}}\mspace{14mu}{point}\mspace{14mu}{for}\mspace{14mu}{needle}\mspace{14mu}{insertion}\mspace{14mu}{arm}}}\end{matrix} & (4) \\{R_{NEEDLE} = {{R_{0}{R\left( \overset{\rightarrow}{\alpha} \right)}} = {{orientation}\mspace{14mu}{of}\mspace{14mu}{needle}\mspace{14mu}{arm}}}} & (5) \\{\overset{\rightarrow}{\alpha} = {{vector}\mspace{14mu}{representation}\mspace{14mu}{for}\mspace{14mu}{small}\mspace{14mu}{rotation}}} & (6) \\{F_{LUS} = {\left\lbrack {R_{LUS},{\overset{\rightarrow}{P}}_{LUS}} \right\rbrack = {{pose}\mspace{14mu}{of}\mspace{14mu}{LUS}\mspace{14mu}{sensor}}}} & (7) \\{V_{TARGET} = {{position}\mspace{14mu}{of}\mspace{14mu}{target}\mspace{14mu}{wrt}\mspace{14mu}{LUS}\mspace{14mu}{sensor}}} & (8)\end{matrix}$

Then the basic constraint is that the needle axis (which is assumed forthis example to be the {right arrow over (Z)} axis of the needle driver)should be aimed at the target lesion, which will be given by F_(LUS){right arrow over (V)}_(TARGET). One metric for the aiming directionerror will be:

$\begin{matrix}\begin{matrix}{{ɛ_{AIMING}\left( \overset{\rightarrow}{\alpha} \right)} = {{\left( {R_{NEEDLE}\overset{\rightarrow}{z}} \right) \times \left( {{F_{LUS}{\overset{\rightarrow}{v}}_{TARGET}} - {\overset{\rightarrow}{P}}_{TROCAR}} \right)}}^{2}} \\{= {{\left( {{R\left( \overset{\rightarrow}{\alpha} \right)}\overset{\rightarrow}{z}} \right) \times {R_{0}^{- 1}\left( {{F_{LUS}{\overset{\rightarrow}{v}}_{TARGET}} - {\overset{\rightarrow}{P}}_{TROCAR}} \right)}}}^{2}}\end{matrix} & (9)\end{matrix}$which can be approximated as a quadratic form in {right arrow over (α)}and converted to a virtual fixture using the method described above.Similarly, if the position of the needle tip is {right arrow over(P)}_(TIP), the penetration depth beyond the LUS target will be givenby:ε_(BEYOND)=(R ₀ R({right arrow over (α)}){right arrow over (z)})●(F_(LUS) {right arrow over (v)} _(TARGET)−{right arrow over(P)}_(TIP))  (10)which can easily be transcribed into a virtual detent or barrierpreventing over-penetration. Alternatively, a simple spherical attractorvirtual fixture can be developed to minimize ∥F_(LUS) {right arrow over(v)}_(TARGET)−{right arrow over (P)}_(TIP)∥.

In process 705, the processor 102 determines the needle tip position asit moves towards the target lesion, and in process 706, the processor102 determines the distance between the needle tip position and thetarget lesion. The needle tip position may be determined from the slavearm kinematics and/or through visual tracking in the camera image.

In process 707, the color of the lesion or some other object in thedisplay changes as the needle tip gets closer to the target. Forexample, the color may start off as blue when the needle tip is stillfar away from the target, and it may change through color spectrum sothat it becomes red as it nears the target. Alternatively, a bar graphor other visual indicator may be used to give a quick sense of thedistance.

In process 708, a determination is made whether the distance has reacheda threshold distance (usually specified as some distance close to oreven at the surface of the target lesion). If the threshold has not beenreached, then the method loops back to process 705 and continuallyrepeats processes 705-708 until the threshold is reached. Once thethreshold is reached, in process 709, a 90 degree view of the cancerousstructure and the approaching needle is shown in a picture-in-picturewindow of the master display 104. The method may then go back to process705 and repeat processes 705-708 as the needle penetrates the cancerousstructure or withdraws back to its start position.

Virtual fixtures, along with other objects, may be defined ormanipulated through an interactive user interface at a surgeon consoleas more fully described below.

Interactive User Interface

Overview

Robotic surgical systems allow a surgeon to operate in situ. Thebenefits of non invasive surgery are well documented and continuingimprovements in laparoscopic surgery are advancing the medicalprofession in a new and exciting direction. One of the many challengesof laparoscopic surgery is working within the confined space of a bodycavity. Surgical instruments, endoscopes, ultrasound probes, etc. needto be directed with precision and celerity, or risk complications fromaccidental tissue damage and extended surgery times. Thus robot assistedlaparoscopic surgery may benefit from an interactive user interface thatprovides a unified assistive environment for surgery. The interactiveuser interface integrates robotic devices, preoperative andintra-operative data sets, surgical task models, and human-machinecooperative manipulation. A surgical assistant workstation (SAW) forteleoperated surgical robots can enhance the capabilities ofrobot-assisted laparoscopic surgery by providing fully integrated imageguidance and data-enhanced intra-operative assistance to the surgicalteam and to the surgeon in particular.

Master tool manipulators (MTM) (e.g., master tool manipulators 107-108illustrated in FIG. 1) are input devices of a surgical console (e.g.,surgeon console C illustrated in FIG. 1) that constitute the primarymeans of input and control for the surgeon. Details of a master toolmanipulator are described in U.S. Pat. No. 6,714,939 entitled MASTERHAVING REDUNDANT DEGREES OF FREEDOM, issued on Mar. 30, 2004 toSalisbury et al. which is incorporated herein by reference.

The master tool manipulators (MTMs) can be switched to operate indifferent modes. U.S. Pat. No. 6,459,926 entitled REPOSITIONING ANDREORIENTATION OF MASTER/SLAVE RELATIONSHIP IN MINIMALLY INVASIVETELESURGERY, issued on Oct. 1, 2006 to William C. Nowlin et al.incorporated by reference, provides further details as to how the mastertool manipulators (MTMs) (also referred to herein as master inputdevices) can be switched to operate in different modes.

In a following mode, the patient-side slave manipulators (PSMs) (alsoreferred to sometimes as robotic arms) follow the motion of the mastertool manipulators and are teleoperated. That is, the MTMs may couplemotion into the patient-side slave manipulators. A third patient-sideslave manipulator (PSM-3) can be activated by tapping the clutch pedal.This allows the surgeon to toggle between PSM-3 and either PSM-1 orPSM-2, depending on which side PSM-3 is positioned.

In a master clutch mode, the master clutch pedal is depressed and thesystem is taken out of following mode. The PSM motion is no longercoupled to MTM motion. During surgery, this allows the operator tore-center the MTMs within their range of motion, and thus increase thesurgical workspace.

In a camera control mode, the camera clutch pedal is depressed and thePSMs are taken out of following mode and control is transferred to theendoscopic control manipulator (ECM) for camera repositioning.

The SAW framework adds another alternative mode (referred to asmasters-as-mice mode) for the MTMs that overlaps with master clutchmode, allowing the surgeon to interact with the SAW graphical userinterface (GUI). In this mode, each MTM operates as a 3D mouse, suchthat it can be used to position a graphical cursor overlaid on thestereo display console, while gripper open/close motions are used toemulate click and drag operations. In this way, the surgeon is able tointeract with graphical objects and menus displayed by the SAWapplication. This mode is called a masters-as-mice (MaM) mode.

When using the surgeon or surgical console, the master tool manipulatorsare used as input devices for the graphical user interface within thesurgical console. The MaM mode overlaps with the existing master clutchmode of the surgeon console in the following way:

Process MTM Event(Event) 1 if Event == MASTER CLUTCH PRESSED 2 then InitMTM Pos = GetMTMPos( ) 3   Wait(3 seconds) 4   ClutchState =GetMasterClutchState( ) 5   MTM Pos = GetMTMPos( ) 6   if (ClutchState== PRESSED) and ((MTM Pos − InitMTM   Pos) < epsilon) 7    thenEnterSAWConsoleMode( ) 8    else return 9  else returnWhile in Saw Console Mode (MaM and/or GUI modes) the position andorientation of the MTM is used to drive the 3D pointer, while itsgripper handle is used as a button.

A diagrammatic view of a teleoperated surgical system including asurgical assistant workstation (SAW) is shown in FIG. 12. Deployment ofthe SAW framework is application specific. FIG. 12 shows a genericdeployment view that illustrates a number of common components andsub-systems.

A user interface 1202 of a teleoperated surgical system is connected viaa communication network to a SAW 1210. SAW 1210 will support at leasttwo types of video sources, namely: stereo endoscopy 1204 and ultrasound1206. Stereo endoscopy may be provided by two laparoscopic cameras(endoscopes or endoscopic cameras) transmitting independent video imagesto stereo displays 1216, such as at the master display 104 of theconsole C illustrated in FIG. 1 or head mounted displays or visors.Ultrasound 1206 may be an ultrasound probe attached to the end of awristed robotic surgical arm inserted into the surgical site. Stereoendoscopy 1204 may be connected to SAW 1210 by analog or digital video.In addition to analog or digital video, ultrasound 1206 may be connectedto SAW 1210 by network interface. Video images may also be provided by amedical image database 1212 connected to the SAW 1210. The medical imagedatabase 1212 is a source of medical images, models, surgical plans, andother application data. For example the medical images database couldinclude preoperative images or a clinical Picture Archiving andCommunication system (PACS).

Master robot 1208 and slave robot 1214 are research-grade interfacedevices generally with robotic surgical arms operating various surgicaltools. Examples of the master robot 1208 include CISST MTMs and steadyhand Robot. Examples of slave robot 1214 include CISST PSMs, and a snakerobot.

FIG. 13 is a diagrammatic illustration of a surgical assistantworkstation (SAW) system architecture 1300 for teleoperated surgicalrobots. The SAW system architecture 1300 includes multipleinterconnected subsystems, which are briefly described hereafter. Avideo subsystem 1301 provides mechanisms for acquiring and processingstreams of images, including ultrasound and stereo endoscopic video.Such image processing pipelines can be used to implement tool and tissuetracking algorithms. Tool tracking 1302 is a specialized imageprocessing pipeline provided for tracking the positions of surgicalinstruments using a combination of kinematic and stereo vision feedback.

Another subsystem is the calibration and registration subsystem 1303.This subsystem may provide software tools for determining devicecalibration, as well as methods for computing coordinate transformationsbetween data sources (i.e., registration). Such tools may includekinematic calibration, camera calibration, ultrasound calibration,pre-operative and intra-operative image registration, video registrationand overlay, etc.

The data management subsystem 1304 provides means to both import andexport archived application data, including medical images, models,surgical plans and annotations. In its implementation, this subsystemcould accommodate data in various formats, including medical realitymarkup language (MRML), DICOM and clinical PACS.

The communication interface 1305 facilitates interactive manipulationand visualization of 2D and 3D data objects, including medical imagesand video, directly within the surgical console. A 3D graphical userinterface manages user interaction from various input devices (includingthe master tool manipulators MTMs) and renders a menu system andgraphical overlays to the stereo display of the surgical console. A 3Dbrick manager (as opposed to a 2D Window Manager) providesapplication-level widgets and interaction logic. A secondary userinterface, called the staff console, will be provided to support thesurgical interface. This is a conventional 2D interface that is intendedfor planning and monitoring outside of the surgical console.

FIG. 14 shows an illustrative data flow diagram, focusing on the robotapplication program interface (API) and the pipeline for videoprocessing and visualization. This figure also shows the tool trackingand volume viewer subsystems. Although not specifically shown,calibration and registration functions may also be performed.

In FIG. 14, subsystems are shown with arrows illustrating data flowbetween the various subsystems of the SAW 1400. Robot system block 1402and collaborative robot block 1404 transmit kinematic motion data tovolume viewer block 1408 and tool tracking block 1410. Volume viewerblock 1408 also receives preoperative image/model data 1406 from thedata management subsystem 1304 in FIG. 13.

In the video processing/visualization pathways, image data from cameras1426 and LapUS 1428 is captured by their respective image capturemodules, stereo image capture module 1420 and ultrasound (US) imagecapture module 1422. Video image data from the endoscopic cameras 1426is further rectified in the rectification block 1416 before beingcoupled into the stereo processor block 1412 for processing from 2D to3D images. The 3D images of block 1412 are then transmitted to the tooltracking subsystem 1410 and used in conjunction with the kinematic dataprovided by collaborative robot block 1404 to monitor the surgicaltools.

After being captured by the ultrasound (US) image capture module 1422,the LapUS data is transmitted to the image fusion block 1414. The imagefusion block 1414 fuses the ultrasound images with the 3D endoscopicimages that are then coupled into the overlay block 1418. The overlayblock 1418 selectively overlays the graphical user interface and themedical image volume onto the fused ultrasound and endoscopic images.The combined image data including the overlaid graphics and images ontothe fused images is coupled to the rendering block 1424 for renderingonto the hardware display 1430.

FIG. 15 is a logical view of the subsystem architecture of SAW 1500. Arobot manipulator (master device 1501 and a slave device 1502), imagesources (endoscope image source 1504 and ultrasound image source 1505),external consoles 1506 (staff console) and other peripherals 1507(general) are categorized as devices, and as such are interfaced to theapplication framework by means of device interfaces. Thesedevice-specific blocks create a layer of abstraction between externalhardware or software modules in order to present a uniform interface tothe application logic.

The collaborative control block 1508 couples the master and slavedevices together. In a single-slave, single-master configuration, thisblock implements teleoperation control. In general, an application mayinclude multiple masters and/or slaves; therefore, the collaborativecontrol block provides a means to coordinate multiple manipulators. Itcontains a synchronous real-time loop for implementing control systems.

A video processing pipeline is used to implement visual tool/instrumenttracking 1510. The visual tool/instrument tracking block 1510 receivesstate information from the collaborative control block 1508 in order toincorporate kinematic information into the tool tracking algorithm.Exemplary tool tracking algorithms and systems that may be used aredescribed in U.S. patent application Ser. No. 11/130,471 entitledMETHODS AND SYSTEM FOR PERFORMING 3-D TOOL TRACKING BY FUSION OF SENSORAND/OR CAMERA DERIVED DATA DURING MINIMALLY INVASIVE SURGERY, filed onMay 16, 2005 by Brian David Hoffman et al.

The master interaction block 1512 facilitates user interaction with menuwidgets and graphical scene objects represented by the brick manager1514. It provides the interface logic between the master manipulatorsand the brick manager 1514 when in masters-as-mice mode. Typical 2Dwindowing systems use the mouse input to create events (e.g., motion,click, release events) and bind callbacks to these events. The masterInteraction block 1512 provides a similar mechanism for the 3D MTMinputs by querying the state of the manipulators and listening forclutch events. The interaction logic transforms these inputs intopointer motion, button click events and specific behaviors such asobject selection, dragging, rotation, resizing, etc.

The brick manager 1514 is the three dimensional analog of a standardwindow manager, in that it supports 3D user input and interaction with3D graphical objects, such as image volumes and models, markers,annotations and in-situ video streams. The visual scene that ismaintained by the brick manager 1514 is ultimately rendered in stereofor overlay onto the surgical console display. It can be used to provideintraoperative visualization and graphical user interface (GUI). Thebrick manager 1514 renders the fixed/augmented view into an interactivewindow the surgeon can interact with. A display driver 1524 drives imagedata onto the left and right channels of the stereoscopic display.

Application-specific logic is encapsulated in SAW application block 1516and is defined by the application developer within the scope of the SAWapplication framework. Once the “master Interaction” component hasdetermined which widget is currently active, all events will beforwarded to the widget and its logical layer. If the applicationrequires a more direct access to the MTMs, the application will be ableto access the MTM's state and disable the event forwarding from themaster interaction component.

Data block 1518 contains images, text, and other data which can becalled by surgeon via the master interaction block 1512 and SAWApplication Logic 1516.

System calibration is performed in calibration block 1520. Typicalcalibration tasks include kinematic calibration of the robotmanipulators, calibration of the navigation system, ultrasoundcalibration, and model to video registration. Calibration block 1520 mayalso align a video image such as an ultrasound to the coordinate frameof a surgical instrument as seen under an endoscope. Some of thesecalibrations procedures are described further herein.

FIGS. 16, 17, and 18 are logic trees for basic 3D cursor interactions.Two events Move and Grab are diagrammed in more detail in FIGS. 17 and18 respectively.

FIG. 19 depicts the concurrent units of execution in the system. Ingeneral, these execution units are provided by threads (e.g.,multi-threading), rather than by multiple processes. Note, however, thatthe “low-level robot control” may be provided externally (e.g., whenusing the research API). In this case, it would be a separate process,possibly on a separate computer. Similarly, signal and image processingpipelines may be distributed as external processes on separate computinghardware.

Surgical console block 1902 is an interactive intraoperative 3Dgraphical user interface. The GUI may augment the master surgicalinterface for enhanced image visualization and control by the surgeon.Augmentation is accomplished by video overlay of medical volume data oroverlay of live images from a video source such as a LapUS probe orother imaging device. Content specific interactive menus and icons arealso placed on the GUI allowing the surgeon to rotate images, pan, orzoom images, and establish virtual operating boundaries for surgicaltools.

Scene rendering block 1904 is a graphical rendering pipeline responsiblefor stereo visualization and overlay in the surgeon's console. In scenerendering block 1904, video signals from a video source such as anultrasound may be overlaid onto the coordinate frame of a surgicalinstrument operating in the field of view of the endoscope. Video fromthe endoscopes are also processed into 3D images and displayed on thesurgeon console or head mounted display.

Signal/image processing pipeline 1906 is a processing pipeline that isused for video processing such as instrument tracking and image overlayand other signal processing tasks. This pipeline may include theacquisition of images, video, and signals that originate from externaldevices or distributed system components. For some applications,computationally demanding or specialized processing may be performed ona dedicated hardware system. Thus, the signal/image processing pipeline1906 component may also be performed by an external signal processingsystem.

FIG. 20 depicts a hierarchical view of the core SAW software librariesand their dependencies. The bottom rows contain the CISST foundationlibraries, as well as external packages such as Python, LAPACK, and theresearch API. The cisstDevice Interface library includes the abstractbase class for all device interfaces, whether Device Tasks or DeviceWrappers. Specific device interfaces are derived from this class.Similarly, cisstRobot defines generic robot capabilities, whereasrobot-specific implementations are provided by modules such as cisstISI(for the Intuitive Surgical daVinci robot). The figure also showshigher-level functionality such as video processing, instrumenttracking, and collaborative robot control. All of this is encompassed bythe SAW application framework.

In more detail, cisstISI 2002 is a wrapper class that encapsulates ISIAPI functions with cisstlibrary-compatible interfaces and emulatingthese functions for non-daVinci hardware, where appropriate. Wrappersare device interfaces that do not include a thread of execution and are“wrappers” around the device drivers or vendor APIs. CisstStereoVision2004 is an algorithm for managing stereo image pairs and geometry, usedin presenting stereo endoscope images to the surgeon console or headset.Open GL stands for Open Graphics Library and is a standard specificationdefining a cross-language cross-platform API for writing applicationsthat produce 3D computer graphics. The visualization toolkit VTK 2008 isan open source, freely available software system for 3D computergraphics, image processing, and visualization. As previously discussed,brick manager 2010 is a 3D scene manager for the surgeon console similarto a 2D window manager. Block 2012 is the user interface (UI)interaction module. The UI interaction module 2012 is the coreinteraction logic that defines the operation of the user interface atthe surgeon console. This component manages user input from the masterinterface and interprets this input with respect to scene objectsmanaged by the brick manager. Movements of the MTMs in combination withgrip open and close motions are correlated with scene objects such asicons and menus to produce a predefined result.

Examples of the capabilities of an interactive user interface system areillustrated in the following scenarios. These are simplified examplesfor illustrative purposes only. While certain exemplary embodiments aredescribed, it is to be understood that such embodiments are merelyillustrative of and not restrictive on the broad invention, and that theembodiments of the invention not be limited to the specific uses shown.

Image Guidance Using a Laparoscopic Ultrasound Instrument

In this exemplary scenario, a dynamic laparoscopic ultrasound (LapUs)image is overlaid on a tracked LapUS instrument in the stereo endoscopeview provided by the surgeon console of the surgical system.

FIG. 21 is a diagrammatic view of a stereoscopic interface display for asurgeon's console in a minimally invasive surgical system for a user toview images in three dimensions. The display shows tissue 2102 at asurgical site that is within the field of view of an endoscope (notshown). The display also shows an exemplary minimally invasive surgicalinstrument 2104 (e.g., a laparoscopic ultrasound instrument) thatappears to extend into the visual field.

In one aspect, an ultrasound image of tissue is displayed within aninset window 2106 of the display (LapUS inset view). Alternatively, theinset window 2106 may be displayed outside the boundaries of the livevideo image.

In another aspect, a flashlight image window 2108 that shows anultrasound image of tissue is electronically attached to the image ofsurgical instrument 2104 within the display (LapUS flashlight view). Theimages in the flashlight image window 2108 may be live image data(intra-operative images) from a LapUS probe or some other imagingsource. As depicted in FIG. 21, the effect of the flashlight imagewindow 2108 is that it appears attached to the instrument 2104 similarto how a flag is attached to a flagpole. However, the flashlight imagewindow 2108 may be other shapes, and in some aspects is not necessarilyattached to the image of surgical instrument 2104. U.S. Pat. No.6,799,065 entitled IMAGE SHIFTING APPARATUS AND METHOD FOR A TELEROBOTICSYSTEM, issued on Sep. 28, 2004 to Gunter D. Niemeyer, incorporatedherein by reference describes an image shifting mechanism that may beused to facilitate the appearance of the flashlight image window 2108being substantially connected to the LapUS probe.

The flashlight image window 2108 moves as the surgeon moves instrument2104. The image displayed in the flashlight image window 2108 may becomeforeshortened as the surgeon moves instrument 2104, e.g., the surgeonpoints instrument 2104 more deeply into the surgical site. Theflashlight image 2108 may change angle and orientation corresponding tothe movement of the surgical instrument 2104 to indicate the orientationof the ultrasound sensor 301. That is, the ultrasound images slicescaptured by the ultrasound probe may be overlaid into the camera imagesso as to appear as to be emanating in proper perspective from theultrasound sensor. Thus, the effect is of a flashlight that can beshined at various positions within the displayed image to provideenhanced visual information (e.g., the ultrasound image) to a surgeon.

Prior to engaging the SAW, a laparoscopic ultrasound instrument isattached to one of the active patient side manipulators (PSMs) and isinserted through a cannula into the body cavity. The ultrasoundtransducer is calibrated to the surgical instrument. Furthermore,endoscopic video outputs from the surgical system are connected to theSAW, video output from a diagnostic ultrasound device is connected tothe SAW, video output of the SAW is connected to the master surgeonconsole, and the SAW may be connected to the surgical system through anetwork interconnection (e.g., Ethernet). A surgeon operates the mastersurgeon console which displays the live endoscopic images and allows thesurgeon to operate the PSMs via the master tool manipulators (MTMs).

FIGS. 22A-22C are illustrations of a surgery performed while in theLapUS flashlight view mode. A flashlight image window 2108 is attachedto the surgical instrument 2104 as shown in FIG. 21. In this example,the surgical instrument 2104 is a LapUS probe (see laproscopicultrasound probe 150 illustrated in FIG. 3) with a wristed joint 2202for increased degrees of freedom of movement.

In FIG. 22B, a first orientation of surgical instrument and flashlightimage window 2108 are shown. In FIG. 22B, the orientation of theflashlight image window 2108 has slightly changed with a slight changein the orientation of the surgical instrument from that shown in FIG.22A. Note that the flashlight image window 2108 is slightly away fromthe viewer compared to the flashlight image window 2108 illustrated inFIG. 22A. Also note that the video image displayed in flashlight view22B has changed slightly as well due to foreshortening. LapUS probe 2104captures slices of images under the ultrasound sensor 301 in the probehead. Thus, the captured image slices and the flashlight image windowchange as the probe head moves the ultrasound sensor 301 around thesurgical site.

To engage the graphical user interface mode, a surgeon depresses themaster clutch pedal on the surgeon console and closes both master inputdevices (e.g., MTMs) in order to enter a masters-as-mice mode. Themaster input devices may be held closed for a predetermined period oftime in order to enter the masters-as-mice mode in another embodiment.

In FIG. 22C, the graphic user interface (GUI) mode is active. In the GUImode, a 3D pointer/cursor 2212 and a first menu system (including menubuttons 2208 and 2210) may be overlaid onto the camera images of thesurgical site displayed at the surgeon console. Graphical tool icons2204 and 2206 may also be overlaid near each PSM instrument.

In FIG. 22C, various icons in the graphical user interface may beoverlaid onto the images of the surgical site in the display 2200. Anicon may provide information, open a menu system to provide additionalcontrol or functionality, and be context specific depending on whatsurgical instrument 2104 is being used. For example, graphical toolicons 2204 and 2206 indicate a masters-as-mice mode and a graphical userinterface (GUI) mode has been entered for the master input devices.Furthermore, the graphical tool icons 2204 and 2206 adjacent theirrespective instrument may be selected to provide additional informationor further control and/or functionality depending upon the type ofsurgical instrument.

In FIG. 22C, the first menu system including menu buttons 2208 and 2210may be used to further interact with the graphical user interface. Forexample, menu button 2208 may be used to open and close the LapUSflashlight image window 2108 while menu button 2210 may be used to openand close a medical image view mode.

FIG. 22C also illustrates the 3D pointer 2212 overlaid upon imageswithin the display. In this example, the surgeon has moved the pointer2212 over the flagpole image window 2108 in the display with the masterinput devices in the masters-as-mice mode. To show the pointer in threedimensions selecting various surfaces, the size of the pointer may varyas its depth varies in response to the master input devices in themasters-as-mice mode. This may be seen in the comparison of pointer 2212in FIG. 22C and pointer 2304 in FIG. 23.

FIG. 23 depicts a menu system 2302 which may be displayed in response toselection of an icon or menu button by the 3D pointer/cursor 2304 andthe master input devices. The menu system 2302 that is overlaid onto theimages may be context sensitive, such as being responsive to the type ofsurgical instrument 2104.

The following is description of an exemplary method of interacting withthe GUI in the masters-as-mice mode. Other methods may be used tomanipulate the 3D pointer using the master input devices (MTMs) andshould be considered as part of the inventive concept disclosed in thisapplication.

In one embodiment, the surgeon may move the 3D pointer by manipulatingthe primary MTM. Using the MTM, the surgeon moves the 3D pointer 2304over the tool icon (e.g., icons 2201-2202 in FIG. 22C) attached to theultrasound instrument and closes the grip on the MTM to signal click orselect. A pull-down menu 2302 opens and may display options, such asoption 1, a LapUS flashlight view, and option 2, a LapUS inset view.

The surgeon moves the primary MTM to highlight the first option, theLapUS flashlight view. The surgeon releases the grip on the primary MTMand the ultrasound flashlight image (a plane) is overlaid onto thecamera images in a flashlight image window 2108 adjacent the ultrasoundinstrument 2104. When the surgeon releases the master, the menu systemand tool icons disappear, while the ultrasound overlay remains. Theoverlaid ultrasound flashlight image window 2108 moves with the LapUSinstrument, fixed to the coordinate frame of the ultrasoundtransducer/sensor 301.

Alternatively, the surgeon may select the second option, the LapUS insetview. In the LapUS inset view, the ultrasound image is overlaid onto theendoscopic image within an inset window 2106 in the stereoscopic displayat the surgical console. The LapUS inset window 2106 may be resized byusing the MTMs. The LapUS inset window 2106 may also be moved todifferent positions within the display on the master console.

By overlaying a GUI over live images from the endoscope and furtheroverlaying ultrasound images captured by the ultrasound instrument ontothe live images, the SAW fuses graphical objects with physical objectsin a physical coordinate frame.

FIG. 12 shows an illustrative data flow diagram focusing on the robotAPI and the pipeline for video processing and visualization. This figurealso shows the tool tracking and volume viewer subsystems. Although notspecifically shown, calibration and registration functions may beperformed.

Image Guidance Using a Medical Image Overlay

In this example, a medical image volume is opened at the surgicalconsole of the surgical system. A medical image volume may be apre-operative image including magnetic resonance images, computertomography images, ultrasound images, positron emission tomographyimages, or other known medical image volumes that may be stored in knownformats. The medical image volume is overlaid onto live endoscopicimagery displayed on the surgical console.

FIGS. 24A-24D illustrate a step-by-step procedure to open the medicalimage database. The master controller(s) (also referred to as mastertool manipulators or input devices) that control the slave instrument(s)may be used to control a pointer 2212 in the display. The pointer mayappear to operate in three dimensions rather than in two dimensions asgenerally appears on most graphical user interfaces. Since the displayhas a three dimensional appearance, the user interface can be thought ofas having volume (like a brick) instead of being flat (like atwo-dimensional window).

In one aspect the surgeon may move an overlaid ultrasound image or otherpre- or intra-operative image by using one or more master controllers.The overlaid image is on, e.g., a live video image from an endoscope atthe surgical site. The surgeon may move the overlaid image to align withthe live video image of a particular tissue structure. In one aspect,one master controls the overlaid image position, and another mastercontrols the overlaid image orientation. Such image movement control maybe somewhat analogous to that of the endoscopic camera control mode in arobotic surgical system. In these aspects, the overlaid image is notautomatically registered with a tissue structure seen in the live video.

Prior to engaging the SAW system, endoscopic video outputs from therobotic surgical system are connected to the SAW, video output of theSAW is connected to the master surgeon console, and the SAW may also beconnected to the robotic surgical system via Ethernet.

The surgeon operates the master console showing stereo display of liveendoscopic images.

In FIG. 24A, the surgeon depresses the master clutch and enters themasters-as-mice mode activating the GUI mode wherein a 3D pointer/cursor2212 and a menu system 2412 are overlaid onto camera images displayed inthe display device of the surgical console. Graphical tool icons (e.g.,icons 2204 and 2206 in FIG. 22C) may also overlaid near each of the PSMtools (not shown in FIG. 24A).

The surgeon moves the 3D pointer 2212 to a pull-down menu button 2422visible on the overlaid menu system 2412 in FIG. 24B by manipulating theprimary MTM. The button may be marked with text such as “View ImageVolume” or an icon of an image volume. The surgeon may select the menubutton in the GUI by closing the grip on the primary MTM. The pull-downmenu may then open to display a list of predetermined image data setslisted by their unique identifiers. The pull-down menus may be contextsensitive, such as the pull down menu 2302 overlaid onto the live cameraimages from the endoscopic camera as shown in FIG. 23.

The surgeon highlights the desired image volume and releases the grip onthe primary MTM causing an annotated image volume bounding box 2432 tobe overlaid onto the camera images in the display of the surgeon consoleas may be seen in FIGS. 24C and 24D. Depending upon the image volumeselected, a three dimensional image may be displayed in the bounding box2432. Using other menu options 2445A-2445C in another menu system 2444,the surgeon may desire to display a single image slice within thebounding box.

Crosshairs (not shown) may optionally be used to indicate the locationof the origin and eight corners of the image volume in the display.Crosshairs (not shown) may also be optionally used to indicate thelocation of the four corners of an active slice plane in the display.

The surgeon may manipulate the selected image volume by operating theMTMs singularly or conjunctively together in the masters-as-mice and GUImodes. For instance, the primary and/or secondary MTMs may be used topan, zoom and rotate the image volume. U.S. Pat. No. 6,799,065 entitledIMAGE SHIFTING APPARATUS AND METHOD FOR A TELEROBOTIC SYSTEM, issued onSep. 28, 2004 to Gunter D. Niemeyer, incorporated herein by reference,describes an image shifting mechanism that may be used to facilitate themanipulation of image volumes in response to the movement of the primaryand/or secondary MTMs in the masters-as-mice mode.

In one embodiment, the surgeon may move the 3D pointer 2302 over theimage volume 2532 as shown in FIG. 25A and close the grip of the primaryMTM to select the image volume. To pan the image volume, the primary MTMmay be moved to translate the image volume from one position of originto another.

The image volume may be rotated along any axis thereby changing theperspective. Relative motion between the primary and secondary MTM cancontrol the image volume orientation. A surgeon may rotate the imagevolume until the perspective view of the image volume matches theperspective of the live endoscopic camera.

FIGS. 25A-25D illustrate the rotation of an image volume in acounterclockwise direction by manipulating the primary master inputdevice (MTM) and the secondary master input device (MTM) together toform a relative motion there between.

Menu options 2502, 2503, 2445A-2445C of a menu system 2444 overlaid ontothe camera images displayed on the display as shown in FIG. 25D mayallow the surgeon to further manipulate the image volume 2532. Forinstance, in FIG. 25D, the surgeon may opt to remove a surface layer orskin from the displayed image volume 2532 by clicking on a skin menubutton 2502 with the cursor/pointer 2302 in a menu system 2444associated with the image volume. The modified image volume 2532′ isshown in FIG. 25E.

By selecting different menu options, such as sagittal view or axialview, an image slice 2632 of the image volume 2532 defined by a sliceplane 2645 may be displayed within the bounding box 2532 of the imagevolume. Orientation of the slice plane 2645 may be controlled bymovement of the MTMs in the MaM and GUI modes to view different imageslices of the image volume.

In one embodiment, the surgeon moves the 3D pointer over one of thecorners of the slice plane and closes the grip of the primary MTM toselect the slice plane. The secondary MTM may be used to select anotherpar of the slice plane by being positioned over a second corner. Therelative motion between the primary and secondary input devices (MTMs)may be used to control the orientation of the slice plane. The primaryand second MTMs may also be used to reformat the slice plane. The sliceplane follows the motion of the primary MTM. Different slice planes mayalso be displayed as desired by the surgeon.

FIGS. 26A and 26B illustrate sagittal slice planes 2645,2645′ withdifferent positions to form different image slices 2632,2632′ of thesame image volume 2532. Note that the slice plane 2632′ illustrated inFIG. 26B is a slice plane taken further from the center of the skull, ormore laterally than the slice plane 2632 illustrated in FIG. 26A.

In FIGS. 27A and 27B, axial slice planes 2745,2745′ are displayedslicing through the same three dimensional image volume 2532 withdifferent positions to form different image slices 2732,2732′. Similarto sagittal view, the axial slices may be rotated or orientated bymanipulating the MTMs. Also, different slices closer or further awayfrom the crown of the skull may be displayed as the surgeon desires. InFIG. 27B, the surgeon has replaced the skin by selecting the appropriatemenu button 2502 of the menu system 2444 while still in axial view.

To zoom in or out of the image volume, the primary and secondary MTMsmay both be used. With the pointer over the image volume, the primaryMTM is selected by closing its grip to select the image volume. Thesecondary MTM is also selected by closing its grip. A relative distancebetween the primary and secondary MTMs may then be used to control thelevel of zoom of a selected image volume or image slice. Thus, byselecting the image volume and then moving the MTMs to change theirrelative distance of separation, the surgeon may zoom in on or out fromthe image volume to display a desired level of detail.

To exit the image volume mode, a surgeon may re-select an image volumeicon or menu button (e.g., menu button 2210). As depicted in FIG. 28,the surgeon selects the menu button 2210 once again with the pointer2212 over it. This removes the overlay of the image volume 2532 andreturns the initial display of the GUI in the MaM mode, such as shown inFIG. 24B.

To further exit the MaM, the surgeon may press the clutch pedal andclick both master input devices (MTMs) so that the master input devicesmay return to control and couple motion into the minimally invasivesurgical tools of the surgical system.

Note that the forgoing are simplified cases for illustrative purposesonly and should not be considered as limiting the broad inventiveconcepts.

Mentoring

Another embodiment allows surgeons versed in the operation of a roboticsurgical system to mentor another surgeon as a trainee. Two surgeonconsoles are coupled with a single patient-side cart for mentoringpurposes. One surgeon console is designated the supervisor console,while the second is the trainee console. A simplified case of mentoringis now described.

Before mentoring operations are conducted, two surgeon consoles areinterfaced with the SAW. One patient side cart (PSC) is interfaced withthe SAW and stereo endoscopic video output is connected to the SAW. Twosets of stereo video outputs of the SAW are connected, one to each ofthe da Vinci master consoles.

The supervisory surgeon depresses the master clutch pedal on thesurgical console and holds the MTMs steady for three seconds, enteringmasters-as-mice mode. GUI mode becomes active and visible on bothsurgical consoles, a 3D pointer/cursor and menu system are overlaid ontothe surgical consoles. Graphical tool icons appear at each of the PSMtools.

The supervisory surgeon moves the 3D pointer and selects “mentor mode”by closing the primary MTM grip while the pointer is appropriatelypositioned on the graphical menu system. The menu system disappears fromthe trainee console and PSM control is transferred to the trainee. Atelestration menu appears on the supervisory console.

The camera clutch on the supervisory surgeon has shared control of theECM. If both camera clutches are activated, then the trainee consoletakes precedence in order to direct the camera view. Menu options on thesupervisory console allow the supervising surgeon to regain control ofthe PSMs, request control of the fourth arm to control telestration andto overlay pre-operative image volumes.

Further exemplary details of telestration may be found in U.S. patentapplication Ser. No. 11/322,866 entitled STEREO TELESTRATION FOR ROBOTICSURGERY filed on Dec. 30, 2005 by Ben Lamprecht et al., which isincorporated herein by reference.

Virtual Fixtures and the Interactive Graphical User Interface

Another embodiment includes manipulation of virtual fixtures through anInteractive Graphical User Interface. With virtual fixtures, aninteraction mode is formed in which the surgeon shares control of therobot with the computer process. These task-dependent computer processesmay provide assistance to the surgeon by limiting the robot's motionwithin restricted regions and/or by influencing it to move along desiredpaths. U.S. Pat. No. 6,493,608 entitled ASPECTS OF A CONTROL SYSTEM OF AMINIMALLY INVASIVE SURGICAL APPARATUS, issued on Dec. 10, 2002 to GunterD. Niemeyer, incorporated herein by reference, describes further detailsof limiting a robot's motion within restricted regions and/orinfluencing it to move along desired paths.

Virtual fixtures (VFs) may be generally classified as either forbiddenregion virtual fixtures (FRVFs) or guidance virtual fixtures (GVFs)(e.g., haptic guidance). FRVFs allow desired motion only in apredetermined task space, whereas GVFs provide assistance in keeping themotion on desired paths or surfaces. In this architecture, FRVFs aredefined using a fixed number of virtual planes, whereas GVFs can beselected from a predefined set of primitives.

In FIG. 29, an exemplary virtual fixture 2902 in the shape of a boundingbox is illustrated as being placed around a surgical gripper 2904. Inthis example of the virtual fixture 2902, the bounding box is open atthe bottom allowing freedom of movement of the gripper 2904 towards thedirection of the tissue. Lateral movement of the gripper 2904, as wellas upward movement and forward movement of the gripper 2904, arecurtailed by the sides of the bounding box in those directions. Theforbidden region virtual fixtures set predefined limits on instrumentmovement to prevent undesired tissue collision. In another aspect,guidance virtual fixtures may be used to constrain a surgical instrumentto move on a fixed trajectory. For example, a suturing trajectory mayassist the surgeon to perform suturing movements along a curved path.

To enter Haptic Guidance mode the surgeon holds the MTMs steady forthree seconds in order to enter masters-as-mice mode. The GUI modebecomes active and visible on the surgical console and a 3Dpointer/cursor and menu system are overlaid onto the surgical console.Graphical tool icons appear at each of the PSM tools.

The surgeon moves the 3D pointer by manipulating one of the primary MTM,and selects “virtual fixture mode” by closing the primary MTM grip whilethe pointer is appropriately positioned on the graphical menu system. Anew menu system appears on the surgeon console that allows adjustment ofplanes that define the boundary of forbidden regions. In this menusystem, a number of 3D planes are visible to the surgeon.

By using the MTMs the surgeon can grab the available planes aspreviously discussed in masters-as-mice mode (See also medical imageoverlay, similar to adjustment of slice plane image). By moving theplanes to desired locations, the surgeon may create boundaries in whichrobotic surgical tools will not traverse. It may be advantageous todefine boundaries, especially in delicate surgeries to preventaccidental tissue destruction by sharp instruments.

Alternatively, in a GVF mode, the surgeon may choose to add fixtures toa surface plane or a predetermined path. These fixtures may be used toguide the surgical tool along the surface or path, automating certainsteps and also allowing precise placement of surgical tools prior toactual cutting or shearing of tissue.

In one embodiment, context based menus may allow the surgeon to selectfrom a list of predefined virtual fixtures listed by unique identifiers.Using this function may expedite placing boundaries and definingfixtures for routine surgeries and procedures.

After the boundaries or primatives are defined, the surgeon selects the“done” button by closing the primary MTM grip while the pointer isappropriately positioned on the graphical menu system. The surgeonreleases the master clutch and returns to normal operating mode.

Modular Robotic Master/Slave Control System

In another aspect of the embodiments of the invention, the surgicalassistant workstation provides a modular robotic master/slave controlsystem. The modular robotic master/slave control system allows a singlemaster controller (surgeon console) to be used to control two or moredifferent types of patient-side slave manipulators (PSM). For example,in one aspect a single master control station (surgeon console) may beused to control a slave station or robot (patient side manipulator) withrigid surgical instruments (similar to the da Vinci® Surgical Systeminstruments manufactured by Intuitive Surgical Inc.) that may be usedfor abdominal surgery. Alternately, the same master control station maybe used to control a different slave station or robot (patient sidemanipulator) with flexible, snake-like surgical instruments that may beused for laryngeal surgery.

In another aspect, the master control station may be used tocoincidentally control different types of slave stations (differentpatient side carts (PSC) with different patient-side slave manipulators(PSM)) that are coupled to it. For example, a minimally invasivesurgical instrument system may comprise a master console, a first slavestation, and a second slave station coupled together by the surgicalassistant workstation. The robotic arm configuration of the first slavestation is different from the robotic arm configuration of the secondslave station such that different control signals to each are used intheir control. The surgical assistant workstation adapts the masterconsole to interchangeably control either the first slave station or thesecond slave station.

Conclusion

The embodiments of the invention have now been described with somedetail. While certain exemplary embodiments have been described andshown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention, and that the embodiments of the invention not be limited tothe specific constructions and arrangements shown and described, sincevarious other modifications may become apparent after reading thisdisclosure. Instead, the embodiments of the invention should beconstrued according to the claims that follow below.

We claim:
 1. A minimally invasive surgical system comprising: astereoscopic endoscope to capture camera images of a surgical site and afirst tool; a stereoscopic video display device of a surgeon console todisplay stereo video images to a user to view images with athree-dimensional (3D) appearance; a processor coupled to thestereoscopic endoscope and the stereoscopic video display device, inresponse to stored program instructions the processor is configured tooverlay a first tool icon onto the one or more captured camera images ofthe surgical site, wherein the first tool icon is located adjacent thefirst tool without being on top of the first tool, receive a signalselecting the first tool icon; and overlay a first menu system onto theone or more captured camera images of the surgical site based on theselection of the first tool icon, the first menu system including aplurality of first menu buttons associated with the first tool tofurther control the first tool or provide information associated withthe first tool.
 2. The minimally invasive surgical system of claim 1,wherein in response to further stored program instructions the processoris further configured to track a position of the first tool in thesurgical site, and overlay the first tool icon near images of the firsttool in the one or more captured camera images based on the tracking ofthe position of the first tool.
 3. The minimally invasive surgicalsystem of claim 2, wherein in response to further stored programinstructions the processor is further configured to adjust a size of thefirst tool icon in response to a change in depth of the first tool inthe surgical site.
 4. The minimally invasive surgical system of claim 1,further comprising a first menu button in the first menu systemconfigured to overlay intra-operative images captured by the first toolwithin an inset window onto the one or more captured camera images ofthe surgical site.
 5. The minimally invasive surgical system of claim 1,further comprising a second menu button in the first menu systemconfigured to overlay intra-operative images captured by the first toolwithin a flashlight image window onto the one or more captured cameraimages of the surgical site.
 6. The minimally invasive surgical systemof claim 1, further comprising images of a second tool captured by thestereoscopic endoscope; a second tool icon overlaid onto the one or morecaptured camera images of the surgical site, the second tool iconlocated near the second tool; and the processor overlays a second menusystem onto the one or more captured camera images of the surgical sitebased on the selection of the second tool icon, the second menu systemincluding a plurality of second menu buttons associated with a contextof the second tool to further control the second tool or provideinformation associated with the second tool.
 7. A surgeon console of aminimally invasive surgical system comprising: a stereoscopic videodisplay device to display live captured stereo camera images of asurgical site, and a first tool, with a three-dimensional (3D)appearance; a processor coupled to the stereoscopic video displaydevice, in response to stored program instructions the processor isconfigured to overlay a first tool icon onto the one or more livecaptured camera images of the surgical site, wherein the first tool iconis located adjacent the first tool without being on top of the firsttool, receive a signal selecting the first tool icon; and overlay afirst menu system onto the one or more live captured camera images ofthe surgical site based on the selection of the first tool icon, thefirst menu system including a plurality of first menu buttons associatedwith the first tool to further control the first tool or provideinformation associated with the first tool.
 8. The minimally invasivesurgical system of claim 7, further comprising a first menu button inthe first menu system configured to overlay intra-operative imagescaptured by the first tool within an inset window onto the one or morecaptured camera images of the surgical site.
 9. The minimally invasivesurgical system of claim 8, further comprising a second menu button inthe first menu system configured to overlay intra-operative imagescaptured by the first tool within a flashlight image window onto the oneor more captured camera images of the surgical site.